LASSO REGRESSION: ESTIMATION AND SHRINKAGE VIA LIMIT OF GIBBS SAMPLING By

The application of the lasso is espoused in high-dimensional settings where only a small number of the regression coefficients are believed to be nonzero (i.e., the solution is sparse). Moreover, statistical properties of high-dimensional lasso estimators are often proved under the assumption that the correlation between the predictors is bounded. In this vein, coordinatewise methods, the most common means of computing the lasso solution, naturally work well in the presence of low to moderate multicollinearity. The computational speed of coordinatewise algorithms, while excellent for sparse and low to moderate multicollinearity settings, degrades as sparsity decreases and multicollinearity increases. Though lack of sparsity and high multicollinearity can be quite common in contemporary applications, model selection is still a necessity in such settings. Motivated by the limitations of coordinatewise algorithms in such “non-sparse” and “high-multicollinearity” settings, we propose the novel “Deterministic Bayesian Lasso” algorithm for computing the lasso solution. This algorithm is developed by considering a limiting version of the Bayesian lasso. In contrast to coordinatewise algorithms, the performance of the Deterministic Bayesian Lasso improves as sparsity decreases and multicollinearity increases. Importantly, in non-sparse and high-multicollinearity settings the proposed algorithm can offer substantial increases in computational speed over coordinatewise algorithms. A rigorous theoretical analysis demonstrates that (1) the Deterministic Bayesian Lasso algorithm converges to the lasso solution, and (2) it leads to a representation of the lasso estimator which shows how it achieves both `1 and `2 types of shrinkage simultaneously. Connections between the Deterministic Bayesian Lasso and other algorithms are also provided. The benefits of the Deterministic Bayesian Lasso algorithm are then illustrated on simulated and real data. ar X iv :s ub m it/ 11 52 72 2 [ st at .M E ] 6 J an 2 01 5 This paper has been accepted for publication in Journal of the Royal Statistical Society Series B, Statistical Methodology.